2 links tagged with all of: mechanistic interpretability + ai
Click any tag below to further narrow down your results
Links
This paper explores how large language models make decisions during reasoning. It demonstrates that these models often encode their choices before generating text, influencing their subsequent thought processes. The research shows that altering initial decisions can change reasoning outcomes significantly.
- A linear probe can decode whether a model will call a tool from its activations before it generates any reasoning text, sometimes before any tokens at all
- Artificially flipping this early "decision direction" causes the model to switch its tool-use behavior in 7% to 79% of cases depending on model/benchmark
- When steered toward a different decision, the model's subsequent reasoning rationalizes the new choice rather than resisting or correcting it, suggesting the "thinking" is post-hoc justification rather than genuine deliberation
Researchers used a “concept injection” method to compare Claude’s self-reported thoughts with its actual neural activity. They found Claude Opus 4 and 4.1 sometimes detect and control injected concepts, suggesting limited but real introspective abilities that improve with model capacity.
- Anthropic injected known concept vectors into Claude's activations and found Opus 4/4.1 could sometimes notice and identify the "unexpected thought" before it appeared in output, suggesting real internal monitoring rather than post-hoc confabulation.
- Opus 4.1 only caught these injections about 20% of the time, and weaker models barely detected them at all, showing introspection is real but rare and scales with model capacity.
- Retroactively injecting "bread" into prior activations made Claude claim it had intended to say "bread" and fabricate a justifying backstory, indicating it was consulting an internal record of its own prior state rather than just rereading its text.
- Overly strong injections caused hallucinated, confabulated explanations (e.g., mistaking a "dust" vector for a literal speck), showing the effect is fragile and highly sensitive to injection strength.